Goto

Collaborating Authors

 calibration property


No-Regret Learning with Unbounded Losses: The Case of Logarithmic Pooling Supplementary Appendix May 10, 2023 A. Omitted Proofs

Neural Information Processing Systems

We now prove the first bullet. This is a contradiction, so in fact c κ. The first claim of the second bullet is analogous. To do so, we note the following technical lemma (proof below). To prove (#1), we proceed by induction on t.



Blackwell's Approachability for Sequential Conformal Inference

Principato, Guillaume, Stoltz, Gilles

arXiv.org Machine Learning

Conformal inference [Vovk et al., 2005] provides a general procedure for constructing prediction sets with guaranteed coverage, under the assumption that the data are exchangeable. This assumption, however, is often too restrictive: it typically fails in sequential or time-dependent settings such as time series forecasting, where the distribution of observations may shift over time. To address this issue, Gibbs and Cand` es [2021] introduced Adaptive Conformal Inference (ACI), which extends Conformal Prediction (CP) to adversarial environments. ACI adapts to distribution shifts by updating prediction intervals in response to observed outcomes, ensuring that the empirical coverage converges to the desired level. While effective in maintaining coverage, ACI and its extensions generally lack efficiency guarantees-for instance, there is no control over the average length of prediction intervals in adversarial regimes. In this work, we study sequential conformal inference as a repeated two-player finite game and invoke Blackwell's theory of approachability to characterize feasible objectives. Building on this perspective, we design a calibration-based algorithm that ensures asymptotic validity while achieving asymptotic efficiency under mild assumptions. Our approach relies on the notion of opportunistic approachability [Bernstein et al., 2014], which allows the learner to exploit potential restrictions in the opponent's play. We argue that such assumptions better fit the typical use cases of ACI-such as distributional drift or regime switching-than the fully adversarial setting.





No-Regret Learning with Unbounded Losses: The Case of Logarithmic Pooling

Neyman, Eric, Roughgarden, Tim

arXiv.org Machine Learning

For each of $T$ time steps, $m$ experts report probability distributions over $n$ outcomes; we wish to learn to aggregate these forecasts in a way that attains a no-regret guarantee. We focus on the fundamental and practical aggregation method known as logarithmic pooling -- a weighted average of log odds -- which is in a certain sense the optimal choice of pooling method if one is interested in minimizing log loss (as we take to be our loss function). We consider the problem of learning the best set of parameters (i.e. expert weights) in an online adversarial setting. We assume (by necessity) that the adversarial choices of outcomes and forecasts are consistent, in the sense that experts report calibrated forecasts. Imposing this constraint creates a (to our knowledge) novel semi-adversarial setting in which the adversary retains a large amount of flexibility. In this setting, we present an algorithm based on online mirror descent that learns expert weights in a way that attains $O(\sqrt{T} \log T)$ expected regret as compared with the best weights in hindsight.


On Calibration of Modern Quantized Efficient Neural Networks

Kuang, Joey, Wong, Alexander

arXiv.org Artificial Intelligence

We explore calibration properties at various precisions for three architectures: ShuffleNetv2, GhostNet-VGG, and MobileOne; and two datasets: CIFAR-100 and PathMNIST. The quality of calibration is observed to track the quantization quality; it is well-documented that performance worsens with lower precision, and we observe a similar correlation with poorer calibration. This becomes especially egregious at 4-bit activation regime. GhostNet-VGG is shown to be the most robust to overall performance drop at lower precision. We find that temperature scaling can improve calibration error for quantized networks, with some caveats. We hope that these preliminary insights can lead to more opportunities for explainable and reliable EdgeML.